I have adopted a consistent set of definitions. The next step is to find all the points of self-intersection:
To begin, I intersect each distinct pair of lines, and see if the intersections fall on the edges of the polygon. How many calculations will this take? Since the points are connected, the intersections of adjacent segments are already given by the vertices. Define the coincident case in the same way (they intersect at the common endpoint). Then, discarding adjacent edges, Edge [math] a_1[/math] may intersect [math]a_3, a_4, \ldots, a_{n-1}[/math] Edge [math] a_k,[/math] k= 2, 3, ... n-2 may intersect [math]a_{k+2}, a_{k+3}, \ldots, a_{n}[/math] Each intersection time has the form [math]{\rm \frac{a \times b}{a \times c}}[/math], where a,b, c are vectors in the plane. I have then to calculate this many intersection times: (n-3) +[(n-3)/2 pairs of (n-2)] [math] \;\;= {\small \frac{1}{2}} n (n-3),[/math] An [math]O(n^2) [/math] running time, but a very lightweight one. The point here is to give the intersections. Optimization later. Now, to tackle the problem of disambiguating the polygon. The figure in blue is still pretend: GGB does not know how to consistently measure or display its interior. Let us see how this determination can be made. Anticipating my first plan of attack, I have tracked the intersection times of each point, and kept a running list of the pairs of intersecting segments. I believe that the Boolean inside/outside condition, together with the closed path of vertices will immediately provide a procedure. Onward.