The Financial Industry Regulatory Authority said Wednesday that it had fined New York-based Brown Brothers Harriman & Co. $8 million for substantial anti-money laundering compliance failures — the biggest AML-related fine levied by the self-regulator.

The firm’s AML violations include, among other related violations, failured to monitor penny stock trades — some to undisclosed  customers in countries known for bank secrecy — that netted its clients at least $850 million.

BBH’s former global AML compliance officer, Harold Crawford, was also fined $25,000 and suspended for one month.

FINRA says that BBH also failed to sufficiently investigate potentially suspicious penny stock activity brought to the firm’s attention and did not fulfill its Suspicious Activity Report (SAR) filing requirements. In addition, BBH did not have an adequate supervisory system to prevent the distribution of unregistered securities.

The second highest AML-related fine was ordered way back in January 2007, when FINRA — then the National Association of Securities Dealers — fined Banc of America Investment Services $3 million for failing to comply with AML rules in connection with high-risk accounts.

Assessing firms’ AML procedures is one of FINRA’s exam priorities this year.

Brad Bennett, FINRA’s executive vice president of Enforcement, said in a statement that the sanction in this case “reflects the gravity of Brown Brothers Harriman’s compliance failures. The firm opened its doors to undisclosed sellers of penny stocks from secrecy havens without regard for who was behind those transactions, or whether the stock was properly registered or exempt from registration. This case is a reminder to firms of what can happen if they choose to engage in the penny stock liquidation business when they lack the ability to manage the risks involved.”

As FINRA explains, penny stock transactions pose heightened risks because low-priced securities may be manipulated by fraudsters. FINRA found that from Jan. 1, 2009, to June 30, 2013, BBH executed transactions or delivered securities involving at least 6 billion shares of penny stocks.

“BBH executed these transactions despite the fact that it was unable to obtain information essential to verify that the stocks were free trading,” FINRA said. “In many instances, BBH lacked such basic information as the identity of the stock’s beneficial owner, the circumstances under which the stock was obtained, and the seller’s relationship to the issuer. Penny stock transactions generated at least $850 million in proceeds for BBH’s customers.”

FINRA also found that although BBH was aware that customers were depositing and selling large blocks of penny stocks, the firm “failed to ensure that its supervisory reviews were adequate to determine whether the securities were part of an illegal unregistered distribution.”

FINRA notes its Regulatory Notice 09-05, which discusses “red flags” that should signal a firm to closely scrutinize transactions to determine whether the stock is properly registered or exempt from registration, or whether it is being offered illegally. “BBH customers deposited and sold penny stock shares in transactions that should have raised numerous red flags,” FINRA said.

In concluding these settlements, BBH and Crawford neither admitted nor denied the charges but consented to the entry of FINRA’s findings.

Check out FINRA’s Hot-Button Exam Issues for 2014 on ThinkAdvisor.