Yes, you are right that a language difference is not dictated by one's skin color. A person can speak any language or dialect to which he or she has acquired in whatever language community the person lives in. However, the langauge difference can be formed by a social barrier related to race, socioeconomic status, age, gender, or religion. As I have said before, language varieties are formed by geographic and social barriers and we have a real-life example: Jim Crow's law in the south. With this law, it forced the communities to segregate and it reduced the communities' contact with each other. With the lack of contact, it reduced the flow of language between them. So within the communities, they somehow developed their own variety that was particular to their community. So the racial barrier defined the language varieties. It is the same with Black ASL. In the past, white and black deaf children were isolated from each other and they recieved different language input in their own communities. So that's how Black ASL was formed.
Hi, this is Frances adding to Joe's response. I'm talking to his co-researcher, Ceil Lucas, who points out that the policy of segregating white and black deaf children was far from accidental, so there were no schools for black deaf children until after the Civil War. And the segregation continued until long after Brown v Board of Education. There were separate schools or departments in 17 states and the District of Columbia.