After having authenticated a user with OAuth2, it is useful to know how to call a remote third-party API with the user's OAuth2 account.
- You may have noticed that
IndexController,StockProductController,ChartIndexController, andChartStockControllerinvoke underlying service methods namedgather(…). This concept suggests that lookups to third-party providers (Yahoo!) are proceeded. - In
IndexServiceImpl, for example, you can find thegather(String indexId)method:@Override public Index gather(String indexId) { Index index = indexRepository.findOne(indexId); if(AuthenticationUtil.userHasRole(Role.ROLE_OAUTH2)){ updateIndexAndQuotesFromYahoo(index != null ? Sets.newHashSet(index) : Sets.newHashSet(new Index(indexId))); return indexRepository.findOne(indexId); } return index; } - It is really the
updateIndexAndQuotesFromYahoo(…)method that bridges the service layer to the third-party API:@Autowired private SocialUserService usersConnectionRepository; @Autowired private ConnectionRepository connectionRepository; private void updateIndexAndQuotesFromYahoo(Set<Index> askedContent) { Set<Index> recentlyUpdated = askedContent.stream() .filter(t -> t.getLastUpdate() != null && DateUtil.isRecent(t.getLastUpdate(), 1)) .collect(Collectors.toSet()); if(askedContent.size() != recentlyUpdated.size()){ String guid = AuthenticationUtil.getPrincipal().getUsername(); String token = usersConnectionRepository .getRegisteredSocialUser(guid) .getAccessToken(); Connection<Yahoo2> connection = connectionRepository .getPrimaryConnection(Yahoo2.class); if (connection != null) { askedContent.removeAll(recentlyUpdated); List<String> updatableTickers = askedContent.stream() .map(Index::getId) .collect(Collectors.toList()); List<YahooQuote> yahooQuotes = connection.getApi() .financialOperations().getYahooQuotes(updatableTickers, token); Set<Index> upToDateIndex = yahooQuotes.stream() .map(t -> yahooIndexConverter.convert(t)) .collect(Collectors.toSet()); final Map<String, Index> persistedStocks = indexRepository.save(upToDateIndex) .stream() .collect(Collectors.toMap(Index::getId, Function.identity())); yahooQuotes.stream() .map(sq -> new IndexQuote(sq, persistedStocks.get(sq.getId()))) .collect(Collectors.toSet()); indexQuoteRepository.save(updatableQuotes); } } } - In the case of Facebook, Twitter, or LinkedIn, you should be able to find a complete API adaptor to execute calls to their APIs without having to alter it. In our case, we had to develop the needed adaptor so that financial data can be retrieved and exploited from Yahoo!
- We added two methods to a
FinancialOperationsinterface as shown in this code snippet:public interface FinancialOperations { List<YahooQuote> getYahooQuotes(List<String> tickers, String accessToken) ; byte[] getYahooChart(String indexId, ChartType type, ChartHistoSize histoSize, ChartHistoMovingAverage histoAverage, ChartHistoTimeSpan histoPeriod, Integer intradayWidth, Integer intradayHeight, String token); } - This interface has a
FinancialTemplateimplementation as follows:public class FinancialTemplate extends AbstractYahooOperations implements FinancialOperations { private RestTemplate restTemplate; public FinancialTemplate(RestTemplate restTemplate, boolean isAuthorized, String guid) { super(isAuthorized, guid); this.restTemplate = restTemplate; this.restTemplate.getMessageConverters() add( new YahooQuoteMessageConverter( MediaType.APPLICATION_OCTET_STREAM)); } @Override public List<YahooQuote> getYahooQuotes(List<String> tickers, String token) { requiresAuthorization(); final StringBuilder sbTickers = new StringBuilder(); String url = "quotes.csv?s="; String strTickers = ""; if(tickers.size() > 0){ tickers.forEach(t -> strTickers = sbTickers.toString(); strTickers = strTickers.substring(0, strTickers.length()-1); } HttpHeaders headers = new HttpHeaders(); headers.set("Authorization", "Bearer "+token); HttpEntity<?> entity = new HttpEntity<>(headers); return restTemplate.exchange(buildUri(FINANCIAL, url.concat(strTickers).concat("&f=snopl1c1p2hgbavx c4")), HttpMethod.GET, entity , QuoteWrapper.class).getBody(); } ... } - The
FinancialTemplateclass is initialized as part of the globalYahoo2Templatethat is returned with theconnection.getApi()calls ofIndexServiceImpl. - Using this technique to pull (as needed) not only indices and stock quotes from Yahoo! but also graphs, we are now able to display real-time data from more than 25,000 stocks and 30,000 indices.
- The client side is capable of using the provided HATEOAS links that come along with each result element. It uses these links to render detail views such as Index detail or Stock detail (new screens).
Let's understand the theory behind this recipe.
In the context of our application, there is still one refactoring that needs to be explained. It is about historical data and graphs.
The Yahoo! financial API provides historical data. This data can be used to build graphs, and it was initially planned to do it this way. Now, Yahoo! also generates graphs (for both historical and intraday data) and these graphs are quite customizable (time period, average lines, chart or stock's display option, and so on).
We have decided to drop the historical part, which technically is very similar to quote retrieval (data snapshots), to exclusively use graphs generated by Yahoo!
The idea here is to rely on OAuth authenticated users. Yahoo! provides different rates and limits for authenticated and non-authenticated users. Non-authenticated calls are identified on the Yahoo! side by the calling IP, which will be (more or less) the entire CloudstreetMarket application IP in our case. If Yahoo! considers that there are too many calls coming from our IP, that will be an issue. However, if there are too many calls coming from one specific user, Yahoo! will restrict that user without affecting the rest of the application (and this situation can further be recovered by the application).
As you can see, the method-handlers that potentially deal with the financial data of Yahoo! call the appropriated underlying service through methods named gather().
In these gather() methods, the Yahoo! third-party API interferes between our database and our controllers.
If the user is authenticated with OAuth2, the underlying service checks whether the data exists or not in the database and whether it has been updated recently enough to match a predefined buffer period for the data type (one minute for indices and stocks):
- If the answer is yes, this data is returned to the client
- If the answer is no, the expected data is requested from Yahoo!, transformed, stored in the database, and returned to the client
There is nothing planned at the moment for users who are not authenticated with OAuth, but we can imagine easily making them using a common Yahoo! OAuth account.
For the presented recipe, this part is done in the updateIndexAndQuotesFromYahoo method. Our Spring configuration defines a connectionRepository bean created with a request scope for each user. The connectionRepository instance is created from the createConnectionRepository factory method of our SocialUserServiceImpl.
Based on this, we @Autowire these two beans in our service layer:
@Autowired private SocialUserService usersConnectionRepository; @Autowired private ConnectionRepository connectionRepository;
Then, the updateIndexAndQuotesFromYahoo method obtains the logged-in userId (guid) from the Spring Security:
String guid = AuthenticationUtil.getPrincipal().getUsername();
The access token is extracted from the SocialUser Entity (coming from the database):
String token = usersConnectionRepository .getRegisteredSocialUser(guid).getAccessToken();
The Yahoo! connection is retrieved from the database:
Connection<Yahoo2> connection = connectionRepository.getPrimaryConnection(Yahoo2.class);
If the connection is not null, the third-party API is called from the connection object:
List<YahooQuote> yahooQuotes = connection.getApi() .financialOperations().getYahooQuotes(updatableTickers, token);
Once again, we had to develop the actual FinancialTemplate (the Java representation of the Yahoo! financial API), but you should be able to find such existing implementations for your third-party provider.
This section provides a list of many existing open-source Spring Social adaptors that we can use in our projects
The following address provides an up-to-date aggregation of Spring social extensions for connection-support and API-binding to many popular service providers:
https://github.com/spring-projects/spring-social/wiki/Api-Providers
- Yahoo! financial stock tickers: We have prefilled our database with a set of financial references to Yahoo! (stock references and index references), which allows us to point and search for resources that can, for the second time, be updated with real-time data through the Yahoo! APIs. This set of references comes from the great work published by Samir Khan on his blog accessible at http://investexcel.net/all-yahoo-finance-stock-tickers. This XLS data has then been transformed into SQL by us, using a basic text editor and macros.