The United States Census demographic model and terns have evolved over time. For many years, only Black and White residents were worth counting. (See Measuring Race and Ethnicity Across the Decades, 1790-2010" for a graphic on how classification has changed.)
US Census pages include the disclaimer "Past census reports contain some terms that today’s readers may consider obsolete and inappropriate."
Currently, about six percent of the American population is of Asian ancestry. (Source:United States - Census Bureau Profile)
In this screenshot from the 1870 Census, only Chinese residents were officially mentioned, with "Japanese" as a footnote.
(Source: 1870 Census, Volume I. The Statistics of the Population of the United States.. https://www2.census.gov/library/publications/decennial/1870/population/1870a-04.pdf )
"Which color is to be tabooed next?" Cartoon by Thomas Nast, Harper's Weekly, 1882.
Chinese immigrants arrived in the middle of the 18th century; they provided inexpensive labor in California and other western states. Concerns about threats to American jobs, as well as plenty of xenophobia, led to increasingly restrictive immigration policies.
In his acceptance of the 1880 Republican presidential nomination, James Garfield called Chinese immigration "too much like an invasion to be looked upon without solicitude."
For an overview of Asian immigration, with links to documents, see Restricting Immigration from Asia and the Pacific, 1870s to 1950s.