Any way to set color for value in chart and still filter out values (but retain colors)?

Is there any way that I can hard code a color into a chart and still filter out values?

I have 5 divisions, each with a specific color.  I have put them into a stacked column chart and it looks fine.  My issue arises when I try to filter out some of the divisions.  The colors do not stay with the specified divisions.

Ex.)

  Div 1 = Red

  Div 2 = Green

  Div 3 = Black

When I filter out Div 1, this is what I see:

Div 2 = Red

Div 3 = Green

I have tried standard delivered filter panel, and check-boxes using CHART_1.setDataSelection( )

It seems like the color hard coding in the "additional" properties of the chart are not tied to the value, but are determined by sort order.  Any help is greatly appreciated