Wednesday, January 8, 2020

Testing node and branch methods

I have tested the methods in determining nodes and branches. What is left is the main method that calls these methods. The code can be found on:
https://bitbucket.org/shivkiyer/ppe_simulator/src/testing/
https://sourceforge.net/p/pythonpowerelec/code/ci/testing/tree/

And if you would like a full length course on simulating power electronics using Python, check out:
https://www.udemy.com/course/simulating-power-electronic-circuits-using-python/

Here is the code for the tests in command line and web app:



The tests work ok except for a few discrepancies. There is a case in the branch_advance method where an invalid jump does not throw an exception or even return an indication of an error, but rather returns the correct element. For example, while advancing to the right on a branch, the method checks if there is an element to the right of the current element and if that element has already been added to the temporary branch. Legally the advance should occur if the element on the right exists. There could be no jump executed in which case it is simply an element that exists to the right of the last element in the temporary branch. If a jump is executed, the jump direction should be to the right.

However, if the jump executed has a direction of up, down or right, there will be no exception and the element to the right of the current element will be returned. The only time an exception will be generated will be when the jump direction executed will be to the left. The reason this did not cause a major issue was because the methods related to the jump are fairly ok and do not produce wrong jump directions. But in principle, the code is wrong.

The check should be, according to the jump direction, is there an element in that direction? If there is no jump, we could follow some sequence of directions while searching for the next element.

Now that the methods have been tested, the next issue is to test the outer method determine_nodes_branches which calls these methods. At this point, the only was to test this outer method is manually by inputting different circuits. And that doesn't seem like a good way to test as the testing is not at all automatic and how many circuits can I come up with?

This brings me to the next crossroad. I need an automatic circuit builder. This is because things will get more complicated at the next step where I will have to test loops and loop manipulations. I need to generate loops automatically while also introducing errors and bugs in the loops automatically. So, the circuit builder will need to generate circuit topologies only while testing at every stage. For example, I need to start with a simple two node, three branch, two loop circuit and verify that that is generated. Then continuously add nodes and branches automatically to form new loops. The structure should not matter as long as there are a few random jump labels inserted every now and then.

This circuit builder seems very interesting and challenging. The next few blog posts will be dedicated to this.

Thursday, December 12, 2019

Testing jump labels

I finally wrote the first bunch of serious tests with respect to the way the simulator processes circuits. I grouped together a few tests that test the jump labels in a circuit under a class. The code can be found in my bitbucket and sourceforge repos:
https://bitbucket.org/shivkiyer/ppe_simulator/src/testing/
https://sourceforge.net/p/pythonpowerelec/code/ci/testing/tree/

Also, to get a full length course on power electronics simulations, check out my online course:
https://www.udemy.com/course/simulating-power-electronic-circuits-using-python/

The class in the command line interface is:



The functionalities that I am testing for specifically are:
  1. Check whether an element is a component (could also be a wire), a jump label or no connection at all.
  2. A jump must be the extreme element in a branch segment. If it has components on more than one side, that is a violation that throws an exception.
  3. Two jump labels cannot be adjacent to each other - that is a violation that throws an exception.
  4. A jump label cannot be next to a node - a violation that throws an exception.

Most of the tests could be accomplished by a simple assert statement. However, to check if a method throws an exception, I found this block to be most useful:

with pytest.raises(SystemExit):
    ...method...

So, the method or the block of code must throw an error or else the test fails. SystemExit could be replaced by other errors, but in this case, all I do is exit with an error code of 1 and so a SystemExit is all I can check for.

Conversely, if a method or block of code should not throw and error, the test can be written as:

try:
    ...method...
except:
    pytest.fail('This should haved worked')


The class for the web app is a bit different:



The primary reason is that the web application does not exit with an error code when a violation takes place but merely displays the errors to the user on the web browser. So the check is for the result of the methods to be a non null list that would be an error message.

One major advantage of testing was that I found myself investigating code and thinking of possible ways that it could fail. It is a completely different form of coding as opposed to regular development where you are just trying to get the code to work. In testing, you are thinking of ways to break the code. This is a totally different aspect to development that can be fun if you look at it the right way.













Thursday, December 5, 2019

Tested basic spreadsheet operations


The first basic group of tests have been completed. These are tests related to reading circuit schematics and manipulating the elements of the spreadsheet. The latest code can be found in the testing branch in the repositories:


https://bitbucket.org/shivkiyer/ppe_simulator/src/testing/

https://sourceforge.net/p/pythonpowerelec/code/ci/testing/tree/


Most tests gave expected results. The major TODO in this area would be to make the spreadsheets more flexible. Currently, only CSV files can be read and the delimiter has to be a comma. For anyone who is not a hardcore developer, this might be an issue as very few people deal with CSV files on a regular basis. Spreadsheet files are usually .xlsx files or variants depending on the version of Office.

For now, I will continue on testing other parts of the simulator before making logical changes to the code.

Sunday, October 27, 2019

Unit testing of circuit schematic operations - Part I

After setting up the test environments for the command line and the web app in the previous post, I now get started with some serious testing. To begin with, the code for this can be found in the testing branch in either of these two repositories:
https://sourceforge.net/p/pythonpowerelec/code/ci/testing/tree/
https://bitbucket.org/shivkiyer/ppe_simulator/branch/testing

I expanded the testing of csv_element_2D method to the testing of csv_element_2D and csv_tuple_2D as these two methods quite often go hand in hand since one converts a tuple to a string cell position and the vice versa. Here is the code:

There are two types of tests. First, the manual tests are tests where I know there could be a problem and these are the borderline cases - when for example 1Z becomes 1AA or 9A becomes 10A etc. Manual tests can be limited unless you have a lot of time or the function is very complicated and only manual tests are possible. In this case, automatic tests are also possible. For automatic tests, I randomly generate tuples of x (row) and y (column) positions and convert them into string cell positions using csv_element_2D and then convert the resultant cell positions back into tuples using csv_tuple_2D. I chose 20 such iterations and made asserts for each case.

The method can be extended to the Django web app with minor modifications - only the assert statement of pytest will be replaced by assertEqual of TestCase.

Next comes the csv_reader method that takes in a file object which is the .csv file and returns a 2D matrix (lists embedded within a list) representation of the circuit. This is the code:

It is important to write the desired circuit contents to a file. Simply passing the string contents to csv_reader as the argument will not work. This is because when Python opens a file, it creates an iterable object and so you can iterate line by line. However, if I pass a string, the iteration will happen character by character which is not what we want.

I didn't bother with circuit components or circuit topology because what matters here is the reading of the .csv file. A few errors are expected:
  1. If the separator is anything other than a comma, the reading fails.
  2. If one line has an extra character, the reading fails. This is because the expected result is a 2D matrix. This might be ok in most cases, but it might be a good idea to put a catch block in case, some weird spreadsheet software passes through a schematic with unequal line lengths. This should result in a user readable error rather than a syntax error.
  3. The elements are scrubbed internally which means leading and trailing spaces are removed from the elements. This is ok.

Will need to spend some thought on this as spreadsheet software in general are a bit messy and the need to generate .csv files in a particular form may not be obvious to the normal user.

Sunday, October 20, 2019

Starting unit testing in Python Power Electronics

My work as a web developer has introduced me to the wonders of unit testing. No one likes writing tests and neither did I for a very long time. I relied on manual testing like many others. Until I started writing unit tests in Jasmine for my web apps. The number of scenarios that you can test makes your code way more reliable.

Another advantage of testing is that quite often there are changes you would like to make but don't want to make them right now as you might break the code that is working. And in the end these changes never happen as you get too busy with other stuff. Here, test driven development can solve the problem of this inertia. Even if you do not want to change the code itself, test suites can help to totally determine every potential lapse in a block of code that will take away the fear bit by bit of finally getting around to fixing them.

With these noble thoughts, I have finally got over my laziness and decided to write tests for the circuit simulator. As a start, Python comes with the in-built unittest module for writing tests. Django uses this and creates a class TestCase to test API end points. Besides this, Python also has the pytest module that needs to be separately installed.

So, I decided to write unit tests for the command line interface with pytest and for the web app using Django's TestCase class based on unittest. In the beginning, the code for these will be separate. However, with time, the plan will be have one set of tests and also at the same time have one set of code for both web apps.

So, the directory structure for the simulator is:

README
command_line
            --  circuit_solver.py
            --  circuit_elements.py
            --  circuit_exceptions.py
            --  LICENSE.txt
            --  matrix.py
            --  network_reader.py
            --  solver.py

web_app
            -- requirements.txt
            -- simulator_interface
                     -- LICENSE.txt
                     -- manage.py
                     -- simulation_collection
                     -- media_files
                     -- simulator_interface
                                 -- settings.py
                                 -- urls.py
                                 -- wsgi.py
                     -- simulations
                                 -- circuit_solver.py
                                 -- circuit_elements.py
                                 -- circuit_exceptions.py
                                 -- solver.py
                                 -- network_reader.py
                                 -- matrix.py
                                 -- admin.py
                                 -- views.py
                                 -- models.py
                                 -- tests.py


There may be a few more directories and files, but these are the major ones at least for testing. We will create a tests directory inside command_line directory and house the command line app tests there. We will remove the tests.py file inside the simulations directory  inside web_app/simulator_interface and create a tests directory for the web app tests.

To install pytest, just do:
pip install pytest

For the command line app, I decided to get started with testing by writing a very simple test for the function csv_element_2D:



This function takes a tuple/list that represents a row, column position and converts it to a string that represents the cell position. So, [0,0] will be "1A" on a spreadsheet. So, a test for this function in the the command_line app would be:



The test itself if very simple. We import the method from the file network_reader.py that has the method. The test function has to start with test_ for pytest to execute it. The asset method in pytest merely asserts that for the test to pass, the value returned by the function for a particular argument has to be equal to a certain value.

The challenge in getting this test to run was in pytest being able to find the file/module network_reader.py. For this the first block of code above the function had to be written. I had to extract the parent method and insert it into the python path so that Python will look in that directory for the module and import it.

The test for the web app was a bit easier as I didn't have to struggle with path as much. The only challenge was I was trying to use pytest and it would not work as pytest for some reason could not find django module that is imported in the web app modules. The reason I can think of is in Django 2 onwards, the app needs to be loaded and therefore, django is not available until the app is working. For this, I had to go with the default Django TestCase class.



To run these tests. Inside command_line directory, just run the command:
pytest

There are several arguments that can be passed and I will investigate as time goes on. For the web app, inside the directory that contains manage.py, run:
python manage.py test simulations

The app name simulations is important because the tests directory with the test is inside the simulations app directory.


Friday, March 15, 2019

Second failed attempt at linear model of an LCL filter

In my previous blog post, I had described how trying to approximate a differential of Ldi/dt with a linear model is probably impossible. However, there was a flaw in the first linear model that I attempted between the filter capacitor voltage, the grid current and the grid voltage. I had chosen a case where the grid current were always in quadrature to the grid voltage and the filter capacitor voltage has the same template as the grid voltage.

So, to check out if the flaw would result in any improvement, I used a training model where the filter capacitor voltages were different from the grid voltages both by magnitude and by phase. The result was an equally flawed linear model.

vcf = -0.2091*igrid_ref + 1.1028*vgrid

As is evident, this model is even worse that the previous one. Because of the phase angle variation, the model has chosen a larger multiplying factor for the grid voltage of 1.1028 but has tried to compensate with a larger resistance of 0.2091 which is useless in the face of the multiplying factor to the grid voltage.

So, the conclusion is that trying to approximate a differential in this manner by a linear model using linear regression is quite useless. So, I need to step back and figure out what the major challenge with controlling the current through the grid inductor of a LCL filter is.

The transfer function between the inverter and the final grid current is a 3rd order polynomial. Physically, the challenge is in accurately regulating the filter capacitor voltage so as to regulate the grid current to a desired value. The simplest control that can be achieved in this case is controlling the current immediately at the output of the inverter. In that cases, the transfer function between the inverter current and the inverter voltage (or modulation index) is a first order polynomial.

The problem is that inverter output current and the grid current will differ by a current that flows through the filter capacitor. Again, the problem would boil to estimating the current flowing through the filter capacitor. So, to avoid any form of estimation either of current or of voltages, how do we achieve a closed loop control of the final grid current without any synchronous transformation?

The reason why I avoid synchronous transformation is that it transforms the grid current to dc values. This assumes that there are no harmonics are intended to be injected in the grid currents because if the converter is a harmonic filter and is expected to inject harmonics into the grid, the synchronous transformation will have to be performed for every harmonic.

So with these objectives laid out, I will find better ways to control the inverter with advanced concepts of machine learning.

Monday, March 11, 2019

Failed first attempt at linear model for inverter with LCL filter

In the previous blog post, I had attempted to create a linear model to predict the reference for the filter capacitor voltage such that the current injected into the grid will track it's reference. And when I saw the linear model, I had the feeling, it would probably not work. And it didn't.

The linear model that the LinearRegression module with scikit.linear_model gave me was:

vcf_ref = -0.01157 + 0.04176*igrid_ref + 1.0587*vgrid

Seems elegant and when I tried to fit it to the test set, I got a cluster that fell along a linear line. All this seemed wonderful as an application of machine learning to power electronics. But looking at it from a power electronics perspective, what did I get?

vcf_ref = vdc + Remu*igrid_ref + K*vgrid

Where vdc is a dc bias that apparently the linear model feels is needed. Though there should not be a dc bias needed in an ac system unless there is a dc component due to some severe unbalanced harmonic load, this dc bias is fairly small and can be neglected.

The Remu is actually what the linear model approximated the inductor as. The actual equation is:

vcf = vgrid + igrid*R + L* digrid/dt

The linear model has approximated (igrid*R + L* digrid/dt) by Remu (emulated resistance). And this Remu = 0.04176 which is very small given the fact that an inductor is being modeled by this resistor.

The next major problem is the 1.0587*vgrid. We are scaling the grid voltage by 5.8%. And compared to the other two factors, this is the major factor. So really, all that this linear model does to try to create a reference filter capacitor voltage is to scale up the grid voltage by 5.8%. And this is totally inadequate.

Suppose, the current injected into the grid will have to be in-phase with the grid voltage and this would be a requirement for many distributed energy resources. The filter capacitor voltage would need a phase advance with respect to the grid voltage. Or else, it would not be able to produce a grid current in phase with the grid voltage. Because the current through the inductor would be in quadrature and lagging behind the voltage drop across it.

With the above linear model, the filter capacitor voltage would be scaled up by 5% and so in phase with the grid voltage. The voltage drop across the grid inductor would be in phase with the grid voltage and therefore, the current injected into the grid would be lagging behind the grid voltage by 90 degrees. Which is not what we want.

My hypothesis is that what I need is a more detailed and accurate model for the differential operator. And that may need more complicated techniques that a LinearRegression. But before jumping to that, I would like to exhaust any possible solution that linear models have just to gain knowledge.

In all fairness, the linear model that I got from LinearRegression above was a flawed one. Because what I did to train the model was simplistic. To begin with, I had chosen the equilibrium point to be a modulation signal that was the grid voltage scaled by the dc bus:

m = vgrid/vdc

So, the filter capacitor voltage is very close to the grid voltage in magnitude and most importantly in phase as well. To this modulation signal, I had added a random disturbance.

m + mrand*sin(wt)

Here was the mistake - I was giving the random modulation signal perturbation the same phase as the modulation signal. And so, the filter capacitor voltage voltage only changes in magnitude.

And this is exactly why the linear model we ended up with has the major component that scales up the grid voltage by 5%. Because the linear model saw that in a vast majority of cases, that's all there was to it - the capacitor voltage scaled up by small amount and the grid current in quadrature with the grid voltage. So, the linear model assumed this was the ultimate truth and gave that very convenient equation.

So, my second attempt would be to train the linear model making sure that the filter capacitor voltage differed from the grid voltage by not just magnitude by also in phase.

So, check out these two current snapshots in the previous training case:




In both cases, the current is always 90 degrees lagging behind the grid voltage because filter capacitor voltage is in-phase with the grid voltage. So, the linear model thinks this is the norm, this is always how it is.

In contrast, now I am going to add two components to the random shuffle to the modulation index - an in-phase component and a quadrature component. So, the grid current will have a varying phase angle difference with the grid voltage. Below is an example. With this, I am going to try and retrain the linear model. Probably still won't work, but let's see how it changes.