This map published in 1913 clearly shows the layout of the castle, with the buildings of that time shown. The main bailey is North East of the motte (labelled the mount). I think the motte was spread and levelled towards the South East, probably filling a moat between the original motte and the main bailey.
I've added the red circles and arrows to show the numbered camera positions; they are also numbered in each photo caption. The green dotted line I've added is the footpath.
Map by W. Byde Liebert, courtesy of British History Online.

The motte is now only about 4’4” high. I think the bailey was once very probably much taller, and has been spread out towards the road and back towards the main bailey, to form a big level platform. As there’s a 17th Century and later house on the motte (called a “mount” on the map) the flattening out predates the house.

The earliest mention of a castle at Lavendon was as early as 1192, when the occupier was Henry of Clinton. But it seems the castle had been demolished by 1530.

I don’t think any of the stone remains on site, though I took a good look at a stone wall and barn a little further up the road and wondered if I was looking at bits of castle stone. There are quite a few stone cottages and walls in the village, too. It’s possible that the castle was wooden to begin with, and at least partly rebuilt in stone later.

(2) The main bailey, seen from across the old moat. These modern buildings are all on the bailey.

The castle isn’t open to the public; it’s a working farm. But you can walk around the perimeter and get a good impression of the size of it all.

To reach it, take Castle Road, opposite the church, and park somewhere by the last houses on the left; the road is wide enough there but gets narrower. Keep walking up Castle Road until you see a bench on the verge, on your right.

(3) Taken from the public footpath, not far from the road. I think the motte was spread out to the left making a big (and I presume) level platform on the gently sloping side of the hill. The 17th Century house is in the upper centre of the photo.

The public footpath begins there, and takes you through camera position 3, then 2, then, after going over a stile, position 1. Camera position 4 is on the road.

From there you can cross the drive and follow the footpath around the edge of the main bailey. From the footpath you can see the moat and inner bank through the hedge, but even in the winter it’s not possible to get a clear photo.

(4) Seen from the road, this drive rises up to the Western end of the much flattened motte.

When you reach the gap in the hedge the footpath goes right, along the hedgerow.

You are not at the top of the hill where you might expect a castle to be; but it’s a big flat hill and you can see quite a long way even from ground level. A sentry on the top of the bank around the main bailey would see even further; in most directions at least half a mile.

From here the footpath soon bends round to the South and ends up on the Harrold road, where you can walk back to Lavendon. You might like to do a circular walk that includes the castle.

A bit further up Castle Road there’s a bridleway that will take you to the far end of the North East bailey where the castle had another entrance, but there's not much to see. But walking up there will show you how big the castle really was.