Well, they both generally mean "good evening". But you use one when you greet someone, and you use the other when you are saying farewell.
In other words, BONSOIR is in the same category as HELLO, whereas BONNE SOIRÉE is in the same category as GOODBYE.
Let's imagine for a moment that it's 6PM and you're walking down the street. You enter a store to buy something, and the employee says BONSOIR and you reply the same way. They are saying HELLO to you. Then you get what you need from the store and pay on your way out. When you grab your purchases and make to leave the place, the employee will say BONNE SOIRÉE.
So, as you can see in this example, we use BONSOIR when we arrive at the store, and then we use BONNE SOIRÉE when we leave the store.
In other words.
BONSOIR means GOOD EVENING.
BONNE SOIRÉE means HAVE A NICE EVENING.
Do you notice the difference?
The first one is a way of saying hello.
The second one is a good wish you make to a person you're not going to see for the rest of the evening.
Here's another example. When I studied in France, I stayed at the house of a french lady. Every day at 7:30PM, my housemates and I would leave our bedrooms and go downstairs to the dining room to have dinner, and we would greet our hostess saying BONSOIR. And then, if we were going out to a bar or to meet some friends somewhere, she would say to us BONNE SOIRÉE, wishing us to have a nice time for the rest of our evening.
The main thing you must remember is that BONSOIR equals to GOOD EVENING and BONNE SOIRÉE equals to HAVE A NICE EVENING.