There have been strange glitches in the waveform that I could get rid off when I simulated the system at 100 nanoseconds. But then I felt that was way too demanding of such a system which should need 1 or 2 microseconds or maybe even more since both converters are switching at 5 kHz.
My first suspect was of the loop currents and felt that since the loops were being chosen randomly, they need to be rearranged to ensure that the d/dt of the ODEs were minimal. Several iterations and trials later, no change.
So the next suspect was the nodal analysis. Turns out this was the culprit. The main problem with nodal analysis is that impedances are vastly different. A diode when turned on has a resistance of 0.01 ohm while another diode that is off has a resistance of 100 megaohm. So the admittance matrix has elements that have a factor of 1e+10. This makes it prone to error.
One of the reasons for error turns out to be the nodal analysis that occurs after determining freewheeling operation. Apparently this one is particularly the case when an inductor is treated as a resistance when it has negligible current. By removing the check, the problem disappeared. That is an inductor is always treated as a current source in the nodal analysis immediately following freewheeling action.
The simulation is finally working with almost negligible glitches. Need to investigate the above effect a little more before I release the next version and post the next case. Hopefully by the end of the week.
Progress is going slow. But the good news is I have now moved on to multi-inverter systems. Hopefully, the next will be microgrids and renewable energy integration.