Is there any correlation between the social sentiment stream in microblogs and the stock market?

Since I started to investigate in Sentiment Analysis and Opinion Mining last September, I have found many interesting papers. However, only a few of those that I have found are related with stock market, or even other markets in general.

I really think that this is a very challenging and motivating topic. As an example, in Opfine.com we can observe the potential benefits of this kind of approaches. I do not know how it is implemented or what kind of algorithm lays beyond this web, nevertheless, they claim an accuracy of around 97% [1]. Pretty much from my point of view.

I am guessing if, as Bing Liu suggests in his latest works [2, 3], the recall is so important in this kind of algorithms, in contrast with the preservation of the natural distribution of the sentiments. I mean, perhaps is more practical to filter out strange values or individuals which are hard to classify, on behalf of a higher accuracy of clearer individuals. Maybe, this is the reason for a so high claimed accuracy.

A recent article by Johan Bollen, Huina Mao and Xiao-Jun Zeng [4] applies the Profile of Mood States (POMS) to twitter updates, expanded with the Web 1T 5-gram Version 1. This Google dataset contains 1,176,470,663 five-word sequences that appear at least 40 times. The enlarged lexicon for POMS, with 976 terms, is what the authors of [4] called GPOMS lexicon. After their experimentation, they conclude that the accuracy of Dow Jones predictions can be significantly improved with some specific public mood dimensions, but not with others. Among the six identified dimensions (Calm, Alert, Sure, Vital, Kind, and Happy), adding Calm time series enhanced the accuracy to 87.6% in predicting the daily up and down changes in the closing values of DJIA. This article was recently discussed in ReadWriteWeb by Sarah Perez, where she also mentions two interesting webs related with stock market and twitter: StockTwits.com and FINIF Financial Informatics.

Obviously, I have found other recent articles related with sentiment analysis applied to market and politics prediction. However, they are still in my pending-to-read box. Some of the most interesting are [5, 6, 7, 8].

References

1. About Opfine.com. Available at: http://opfine.com/about.jsp
2. Liu, B. (2010). Sentiment Analysis: A Multi-Faceted Problem. IEEE Intelligent Systems 25 (3). Available at: http://www.cs.uic.edu/~liub/FBS/IEEE-Intell-Sentiment-Analysis.pdf
3. Liu, B. (2010). Sentiment Analysis and Subjectivity. Handbook of Natural Language Processing. Available at: http://www.cs.uic.edu/~liub/FBS/NLP-handbook-sentiment-analysis.pdf
4. Bollen, J., Mao, H., and Zeng, X.J. (2010). Twitter Mood Predicts the Stock Market. Arxiv preprint arXiv:1010.3003
5. Das, S.R. and Chen, M.Y. (2007). Yahoo! for Amazon: Sentiment Extraction from Small Talk on the Web. Management Science 53 (9). Available at: http://algo.scu.edu/~sanjivdas/chat_FINAL.pdf
6. Gilbert, E. and Karahalios, K. (2010). Widespread Worry and the Stock Market. Proceedings of AAAI ICWSM'10. Available at: http://social.cs.uiuc.edu/people/gilbert/pub/icwsm10.worry.gilbert.pdf
7. Asur, S. and Huberman, B.A. (2010). Predicting the Future with Social Media. Arxiv preprint arXiv:1003.5699
8. Tumasjan, A., Sprenger, T. O., Sandner, P. G., and Welpe, I. M. (2010). Predicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment. Proceedings of AAAI ICWSM'10. Available at: http://www.aaai.org/ocs/index.php/ICWSM/ICWSM10/paper/viewFile/1441/1852

Hey guy, where are you?

Well, as a matter of fact, I have no time to write a blog, I have never had. I would like to be more active, but it is almost impossible to find time. I will try, but at least I am now also trying to do a bit of microblogging in twitter (@jbtolosa), which is far more realistic...

So, since my last post, I had finished two official european masters from the University of Oviedo (Universidad de Oviedo):

• Master in Web Engineering at the Computer Science Engineering School (Escuela de Ingeniería Informática) in June 2009. Awarded with the Best Academic Record Award (Research Speciality)
• Master in Soft Computing and Intelligent Data Analysis at the European Centre for Soft Computing in July 2010.

Regarding publications, I have also published several papers:

• José Barranquero Tolosa, Jose E. Labra Gayo, Ana B. Martínez Prieto, Sheila Méndez Núñez and Pratricia Ordóñez de Pablos (March 2010). Interactive Web Environment for Collaborative and Extensible Diagram Based Learning. Computers in Human Behavior, 26 (2), pp. 210-217. DOI: 10.1016/j.chb.2009.10.003
• José Barranquero Tolosa, Sergio Guadarrama (July 2010). Collecting Fuzzy Perceptions from Non-expert Users. In Proceedings of the 19th IEEE International Conference on Fuzzy Systems (FUZZ-IEEE 2010), pp. 1-8, Barcelona (Spain). DOI: 10.1109/FUZZY.2010.5584816
• José Barranquero Tolosa, Oscar Sanjuán-Martínez, Vicente García-Díaz, Cristina Pelayo G-Bustelo and Juan Manuel Cueva Lovelle (2010). Towards the Systematic Measurement of ATL Transformation Models. To appear in Software Practice and Experience (published online in Wiley Online Library). DOI: 10.1002/spe.1033

Moreover, since November 2009, I am enjoying a full-time pre-doc scholarship (FPI) at the Artificial Intelligence Center (Gijón, Spain). This grant is funded by the Spanish Ministry of Science and Innovation (MICINN).

After completing my second master, I have started to work in Sentiment Analysis and Opinion Mining, which covers my present activities nowadays. Hence, future posts will be focused on this topic... so starting again with a new topic... no problem, I can deal with it... again!

[Paper] Towards Meta-Model Interoperability of Models through Intelligent Transformation

I have just received the confirmation of a paper submitted to the International Symposium on Distributed Computing and Artificial Intelligence (DCAI'09), organized into the 10th International Work-Conference on Artificial Neural Networks (IWANN'09). This meeting will be held at the University of Salamanca in June 10-12th, 2009.

Special thanks to Vicente García Díaz, Oscar Sanjuán Martínez, Héctor Fernández Fernández and Gloria García Fernández for their support.

Abstract

Models and transformations between models are provided as the core of Model-Driven Engineering, offering reusability of knowledge and processes. In order to establish the basis of future advances in this emerging paradigm, this paper is focused on the principles of meta-models and transformation models. Moreover, the concept of meta-model is becoming an essential artifact for MDE based solutions, thus we have centered our background review in the state of art related with meta-model specifications and model transformation technologies. Our research is aimed at getting a higher degree of interoperability among available meta-model specifications by raising the transformation models to the upper meta-layers. Some conclusions extracted suggest that this is yet an early solution which demands greater efforts in terms of research, development and specification, with many interesting open subjects like design of generic editors for model-agnostic visual modeling, integration of models instances from different meta-models, improvements of the semantic knowledge offered by actual modeling languages or even the evaluation of the applicability of graph transformation techniques towards formal transformation models.

Full Paper

All papers accepted will be published in an special volume of IWANN in Lecture Notes in Computer Science (LNCS Springer Verlag). IWANN is included in the ranking of the best conferences established by the Computer Science Conference Ranking based on the "Estimated Impact of Conference (EIC)", concretely in position 55 among 620 considered (in the Artificial Intelligence field). Therefore the papers will be indexed by CiteSeer.IST, and by the organization Computing Research and Education Association (CORE).

A special issue in the International Journal of Artificial Intelligent Real-Time Automation (Engineering Applications of Artificial Intelligence), from Elsevier, will include a selection of between 10 to 15 papers published in DCAI 2009 (JCR 2007 - 0.762).

[Poster] Login Pattern Proposal

This draft paper is written as part of a master course in Information Arquitecture, including an A3 poster and a presentaion.

Abstract

Login pattern deals with the need of users to sign in, so data stored by them in previous interactions can be used repeatedly in subsequent processes. This fact involves the use of a basic form where the user is prompted for his credentials. We have evaluated the state of art login pattern with a simplified version of the GOMS genetic evolution-based model in order to compare it with our proposal. We focused our efforts on reducing this kind of interaction to its minimal, reorganizing the user inputs in fewer GUI components, so that novice users will require less time to complete the task.

Poster

[PDF] Login Pattern Proposal Poster

Slides

[PDF] Login Pattern Proposal Slides

Full paper

[PDF] Login Pattern Proposal Paper

Endless JavaScript with Dojo

It was one year ago when my interest in AJAX and Dojo was awoken; while looking for some approach to draw geometrical objects within a web browser to evaluate the viability of a collaborative browser-based client for diagram design. However, I have been interested in JavaScript since many more years. It has been always an exciting language for my eyes, despite of its criticized existence.

Today I have received an email from mind42, announcing the end of the beta period. And due to my untiring curiosity I have been looking for more details. My first impresion is: "Wow, what an amazing tool for my mental divergences! and collaborative! and developed with dojo!"

This video shows some of its features:

Looking deeper, I have found this interesting video from Stefan Schuster where he discusses the possibilities of JavaScript development with Dojo:

You can also download the slides of the talk from Stefan Schuster website, or these other introduction slides about Dojo 0.9.

All this stuff have been very inspiring for me, and I am wondering about an abstract idea. It deals with the copcept of a self-defined web environment, where users could extend GUI components with the powerfull features provided by Dojo, like OO development (dojo.declare), namespaces (dojo.require), event handling (dojo.connect) and more on.

This extensions could be designed with the diagram capabilities of the environment itself or with a web-based text editor, monitoring preferred usage for futher research of their usability.

Moreover, it would be feasible to develop a Dojo IDE with the features claimed in Stefan Schuster's talk by means of this environment. Both diagram and text approaches.

[Draft] Management of Conversation-Oriented Languages for Semantic Web Services through Marked Petri Nets

This draft paper is written as part of a master course in Semantic Web, collecting key issues around composition of semantic web services, from the point of view of the representation of formal execution semantics and conceptual models based on marked Petri nets.

Special thanks to José Emilio Labra Gayo, and Juan Miguel Gómez Berbis for their support.

Abstract

This paper reviews the needs of a collaborative web environment to model visual interactive marked Petri nets, focusing on the issues around the definition of semantic web services interaction languages and modeling of instance conversations derived from them. So that, we start describing the traditional requirements for those languages and discussing the background of graph visualization and interaction concerns. Once reviewed all key issues among those broad areas we present our suggested approach, describing briefly a proof of concept prototype and introducing our future work around graph structured semantic information related with conversation-oriented languages for semantic web services composition and mediation.

Full paper

[PDF] Management of Conversation-Oriented Languages for Semantic Web Services through Marked Petri Nets

[Draft] Visualization, Navigation and Edition of Graph Structured Semantic Information

This draft paper is written as part of a master course in Semantic Web, collecting key issues around graph visualization and interaction, from the point of view of the representation of semantic information based on RDF and OWL.

Special thanks to José Emilio Labra Gayo for such great references, like the graph survey by Herman et al. and the article by M.C. Schraefel and A. Karger.

Abstract

This paper focuses on the concepts related to the visualization, navigation and edition of semantic data which can be represented in graph structured formats, widely used in the Semantic Web; like RDF and other extensions as RDFS or OWL. First of all, a background introduction presents concepts like size, planarity, predictability and time complexity related to graph visualization. Later on we offer an overview of available layout algorithms and interaction key issues. And finally, we describe briefly a proof of concept prototype and introduce our future work around graph structured semantic information tools.

Full paper

[PDF] Visualization, Navigation and Edition of Graph Structured Semantic Information

IADIS Interfaces and Human Computer Interaction Conference 2008 (IHCI)

I have just received the acceptance of the paper:
Web Environment for Collaborative and Extensible Diagram Design

Paper is proposed to be in the proceedings of IADIS Interfaces and Human Computer Interaction Conference 2008 (IHCI) as "Reflection Paper" . However, I had to renounce to publish it due to personal issues, economical mayorly. I also think that it can be improved a lot, as can be realized reading reviewers comments.

Somme positive comments for the paper are:

• "The paper has reasonable technical quality, and clear and consistent diagram and use of technical language"
• "The architecture is based on newly technologies and use well-known patterns"
• "The subject is relevant: Collaborative web-based tools demand a lot of research and implementation work"
• "An excellent idea, very promissing but a lack of credibility as regard to the implementation and the evaluation"
• "Very interesting key issue"

Also, some weak points were remarked by reviewers:

• "This is more a position paper than the result of implementatory work"
• "The lack of any sort of formal discussion or results, makes it hard to see the real world value of the model proposed"
• "Scenarios could be used to improve the explanation, even using hypothetical situations/applications"
• "There is not a proof of concept"
• "In addition, the introduction needs to be expanded to include an overview of the literature on collaborative web-based systems and how is the main differences/similarities with respect to other existing tools (e.g. Google docs)"
• "Unfortunately, there is no evaluation that measures the impact of the system on actual users"
• "It seems unusual to have a paper written in English with references written in any other language"
• "You claim as key words psychology and data mining but you don't adress these questions seriously"
• "Your bibliography could be broadenned"
  

[Draft] Web Environment for Collaborative and Extensible Diagram Design

This is the first paper in which I collaborate as principal author, with the support of José Emilio Labra Gayo, Sheila Méndez Núñez and Ana Belén Martinez Prieto.

Special thanks to Lucía Méndez Núñez for her review of the translation from Spanish to English.

The paper addresess the need of a web environment for colaborative diagram design, with broad posibilities of functional extensions, configuration and personalization by users. The idea grows from my end degree project, but with some new improvements in order to check some metrics about colaborative diagram tools usability, mainly UML, as well as psycological studies of software engineering.

Abstract

There is a growing need for development tools ubiquity, due to the portability and availability problems concerning desktop applications. In this paper, we suggest an approach to avoid any further download or installation. The main goal is to offer a collaborative and extensible web environment which covers a series of domains highly demanded by different kinds of working groups, in which it is crucial to have tools which facilitate the exchange of information and the collaboration among their members. The result would be the development of one or several diagrams accesible from any geographical location, independently of the device employed. By means of this environment, it will be possible to do research on the usability of collaborative tools for design diagrams, mainly UML, as well as research on the psychology of software engineering, assessing the results coming from the employment of methodologies, techniques, paradigms, patterns, etc. Both at an individual and at a collaborative development group level.

Full paper

[PDF] Web Environment for Collaborative and Extensible Diagram Design

[PDF] Entorno Web para el Diseño Colaborativo y Extensible de Diagramas