This is the classic problem of maximum volume. I have created the polygon needed to illustrate the problem and use a slider to allow the students to easily adjust the size of the cutout. I also allow them to step through to see the function for the volume of the box in terms of the cutout size. We then ask them to plot the graph of that function on the Graphics 2 window and add the point on that function that corresponds to value of the cutout size chosen on the slider.
If your students have been exposed to Differential Calculus, you can easily have them add the derivative of that function to the Graphics 2 window, and then find the zero that function to get the optimal solution.