The patterns created by antlion groups are emergent: they don't exhibit top-down structure like a highly regular tiled or even consistent polymorphism across trials.
However, the antlions did cluster somewhat (remaining close to eachother despite available space, in some cases) but regardless maintained sufficient area to capture food, either of the cannibalistic or regular sort.
These patterns likely developed, at least in the short terms these antlions were studied, by slow movement of the pits across the trial area, either by live migration or abandonment of old pits (which often occurred).
The Voronoi diagrams are the primary source which exhibits these traits: scaled down to the window of the trial area which antlions populated, the area claimed by each individual antlion is somewhat consistent, explicable by a selfish algorithm: each antlion wants to optimize its area of ant capture (represented by ``claimed'' regions on the Voronoi diagrams), so the area was shared about equally by the group.
Also, average distance to nearest neighbor decreased with lesser trial area: from 5--6cm on average in the 33x32cm trial down to 3--3.5cm in tho 8x7cm trial, the graph in Figure 3 demonstrates a clear correlation, with a notable (but inconclusive) p-value of about 8\%, between territorial area and total area.
Additionally, compensatory behaviors were exhibited which further managed the population: cannibalism and reclusion both prevented surface overpopulation (because when two antlions were too close, one or the other usually occurred)

On the scale of individual pits, antlions optimize for energy. Unrelated to their partners' pits size, antlions typically size their pits to capture ants.
Weekly feedings helped maintain the natural analogue to scarce ant feedings, so the antlions had to create their pits as determined by the density of the environment (simulated by a small area, which antlions readily detected despite their blindness by extensive trails created in the container).
This caused them to create significantly smaller pits (so much so that at about .8cm deep and .8cm wide, measurement errors became excessively significant) in smaller containers (in terms of depth and width) because the antlions were aware that ants would, regardless, fall in rather than survive throughout the antlion colony.
This is in contrast to the 33x32 where none of the antlions formed pits shallower than 1.1cm and one pit was 4.2cm wide.
Furthermore, large pits may have become an unnecessary aggression or warning mechanism because, in order to preserve the larvae, the species would require sufficiently clear land that a group could populate the surface fully without unintentionally increasing cannibalism rates.