Formal Trust Assessment with Confidence Probability
Formal Trust Assessment with Confidence Probability
KSII Transactions on Internet and Information Systems (TIIS). 2015. Feb, 9(2): 830-842
Copyright © 2015, Korean Society For Internet Information
  • Received : May 12, 2014
  • Accepted : January 21, 2015
  • Published : February 28, 2015
Export by style
Cited by
About the Authors
Mahir Kutay
Department of Computer Engineering, Yasar University Bornova, Izmir, 35100 - TURKEY
Tuncay Ercan
Department of Computer Engineering, Yasar University Bornova, Izmir, 35100 - TURKEY

Trust and trustworthiness of web services and organizations is calculated as scalar values. But there is still a certain level of risk for the overall reliability of this value. In this article, we focus on calculating trust values as intervals between upper and lower bounds based on predefined confidence values through an additional confidence probability. This will give us a more realistic approach to the trust assessments between individuals and organizations. We also developed a web-based software tool, TAST (Trust Assessment Software Tool) that collects the web services’ evaluation of different customer groups for similar organizations through the user interface and calculates the trust intervals for predefined and previously selected confidence values. Our model uses a weighted calculation of mean and variances of customer groups in specific periods and analyses the total and incremental trust of different customer groups.
1. Introduction
W ith the increasing popularity of web services, the trust and trust-based issues have begun to gain more importance. Organizations need to collect feedback from customers to improve their services based on trust dependent practices because customers preferring business with companies they trust. Trust is related to confidence in something that can be the person, the environment or the process to be trusted or whatever it is depended on the terms of social, technological and biological considerations [29] . From the point of organizations, the web services they offered is their first chance to create a reliable impression with their customers. From the individual customer perspective, organizations also need to assure customers of their privacy when they interact with their website by showing that the company services are to the highest standards that will make them more trusted and reliable. Previous customer reviews provide personalized online experiences and create trusted relationships with new customers resulting in increased customer engagement and transactions.
Trust-based services include algorithms that the information network claims its sources are reliable. The importance of these systems grows tremendously with the adoption of social media facilities and software services of cloud computing providers. Ensuring all these services provided by service providers are efficient, easy and safe, customers trust them and continue to use the services and even help them find new users by recommending to people around them. In the world today, online recommendations are key features in the evaluation of different web services and incorporate trust relationships [30] . They also help build a robust trust-based recommendation system to generate personalized or organizational recommendations. If the organizations establish their trust relationships, according to a previously calculated trust value and use it with different services in their environment, this helps both the creation of an appropriate trust relationship and minimizes the risks inherent in mutual shares of two organizations. Marsh in [29] is one of the first researchers to introduce a computational model for trust values and represented them as subjective real numbers between the range -1 and +1 or literally blind trust and complete distrust. He also discussed the benefits and limitations of his approach. Wu and Weaver introduced fuzzy logic into the definition and evaluation of trust and provided a formal representation of fuzzy rules to handle the uncertainty in trust management [37] .
A reasonable trust value evaluation is complex and multidimensional, since it includes various relationships among different entities like organizations, customers, employees and managers and is affected by the effectiveness of the technology utilized. It can also include a vertical or a horizontal approach for the same or separate groups of entities. Trust plays a central role in facilitating these relationships [31] . Collecting trust data is based on how customers/recommenders/users interact with the organizational (online or offline) tools. But they guide and recommend others to buy goods or use the services provided by the organizations. The information comes explicitly from ratings, tags, reviews, or implicitly from how much money and time they spend and can be used to select, filter or sort items. The recommendations may be by the preferences of different users [1] .
A lot of the previous trust and trustworthiness assessment models in literature have been calculated as scalar probability values. Limited to our knowledge, there is no previous study of how much these trust values can be relied upon. In this paper, we focused on calculating trust and popularity values with confidence probability. We evaluated the scalar trust values with a confidence value previously selected by the user and obtained trust intervals between upper and lower bounds and this results in a more meaningful and realistic approach. The main components of our methodology are shown in Fig. 1 . Dotted blue processes also highlight the contributions we made.
PPT Slide
Lager Image
Proposed trust assessment methodology
We took a survey on the literatures on trust, trust assessment, reputation and recommender models across diverse disciplines from both theoretical and practical view. A significant number of examples of these models have been pointed out in the next section. We tried to integrate our framework across the literatures surveyed to build a more robust computational model for overall trust evaluations. Our model has the following contributions:
• defines trust in a specified context-set through the web-based survey data
• replaces scalar trust calculations with real-number intervals
• utilizes an additional confidence probability value in trust calculations
• uses the software application (TAST) we have developed
This article is organized as follows; the next section provides a detailed background on the related work for the assessment of trust in open systems. Section 3 presents our proposed model. How the TAST program created by us, works and calculates the necessary values for trust assessment that is explained in Section 4. Section 5 concludes and points out the directions for the future.
2. Related Work
Trust is an important actor with its multidimensional features and helps resolve every risk issue between individuals and organizations. The authors in [2] presented their findings about the trust relationship between the company and suppliers for a vehicle manufacturer. Their research contributed to exploring the constructs of mutual and interactive trust where one's trust is increasing the other's trust is decreasing in the intra-organizational and inter-organizational relationships.
Recommender systems use different techniques to produce individual recommendations for the requirements of the organization [3] . In general, recommender systems are based on one of two methods [4] . The content filtering approach: that creates a profile for each product or customer [5 - 6] . The alternative method is called collaborative filtering [7] . Collaborative filtering relies only on the assumption that similar users share similar thoughts by analyzing the rating histories of a group of recommendation partners [8] . The content filtering method is more successful with new products to new customer relationships.
For numerical representation of system trust some different metrics like binary [9] , discrete [10] and continuous as percentage [11] , probability [12] or subjective probability [13] can be used. First research studies focus on policy and reputation based trust values. Authors in [14] presented the problem of reputation-based trust management for data management and the semantic level. Their approach did not require central control and evaluated trust by computing an agent’s reputation from its former interactions with other agents. Years later, studies shifted to content and metadata based trust evaluation topics. Jacobi, Kagal and Khandewal identified these two categories and proposed a semantic meta-modeling that uses trust ontologies to assign trust values to data sources on the Web [15] .
Some studies focus on tools that can help in experimental studies achievements [16] . Govindan et al. explained a software tool “ProNet” that acts on the received information item to determine the information trust, node-level trust and sequence-level trust. In [17] , the authors proposed an agent model with a machine learning algorithm using the previous transactions to assess the trustworthiness of a potential transaction. This model could distinguish successful transactions from unsuccessful ones and makes analysis of the potential transaction and previous transactions.
A comprehensive trust assessment methodology was presented in [18] . It also included the results of service testing, and formal analysis of service properties and reputation-based ratings. Later, Rettinger, Nickles, and Tresp calculated the context sensitive trust using statistical relational learning in the form of the Infinite Hidden Relational Trust Model (IHRTM) and evaluated his studies empirically on the user-ratings gathered from eBay [19] .
Pasternack and Roth presented three new trust metrics for an information source as truthfulness, completeness and bias scores bring a solution to misleading results based on the scalar values of the current computational trust systems [20] . They also assessed trustworthiness with these three rather than a scalar value. They thought that the user himself was essential in calculating the trustworthiness of a source in addition to his prior knowledge [21] . Caverlee, Liu, and Webb also studied three key factors for trust establishment as incorporating personal user feedback, distinguishing user relationship quality from trust and tackling user behavior in [22] . Authors in [23] proposed a composite trust management that combined the vertical and horizontal trust metrics between individuals and institutions.
Recent studies (some are still in press) present new trust algorithms to select collaborators by calculating their trust measures according to the expected performance of agents by analyzing their previous experiences in [24] . In this article, the authors also present another framework to detect and filter out dishonest feedbacks by using personal experience and a variable tolerance threshold.
To the best of our knowledge, the problem of how to determine service trustworthiness in different environments has been addressed in a manner that is satisfactory with its predefined metrics and methodologies. Our paper makes a contribution at this point to interrogate the trust assessment according to the specified context that automatically narrows the scope and allows us to obtain more likely results. How a predefined context-set including recommenders and object features can be organized has been explained in section 3.1. In the existing literature, trust values are simple scalar or probability scores which do not give any adequate opportunity to challenge their values [32] [33] [34] .
Addition of importance parameter to trust calculations in real-number intervals make a significant difference in our work. Pre-selected confidence interval is an indicator of the calculation’s precision that shows how close it is to the original estimate. Confidence probability is a predefined real value in the interval [0,1] and taken as higher than 0.80 in our model. Importance value is used for reflecting the consideration of users before the construction process of the popularity matrix. It increases the discrimination of user preferences and contributes to obtaining more realistic results in this paper, although not used by previous studies [35] [36] .
Our model can differentiate the ratings of contexts by customer groups in selected time intervals. Comparison of trust variations for the service provider itself and with its competitors is also a great advance. Compared to the papers cited above, our research is based on adding the manually updated confidence value to the trust calculations and evaluates the trust value with upper and lower bounds rather than having one scalar value that is open to current discussions. These probability values based on predefined confidence give a more meaningful and realistic approach.
3. Proposed Model
We introduce our trust assessment model to evaluate the trustworthy features of a system. We made necessary changes to our TAST program which enables to select user groups and calculate trust values with confidence probability. Organizations may have a large span of activities and each independent activity can be named as an individual system. In our model, systems are described by predefined contexts based on the system’s functional activities. This feature provides a more generic framework for the assessment of trust that can easily adapt to any other real-world applications. System trust is the assessed value obtained by processing web-based survey data using our model. System trust does not necessarily mean that a customer have overall trust in the organization.
The sophisticated TAST software can obtain the collected data in real time and make assessments of popularity and system trust values of hotels, hospitals, banking systems or almost every system using web-based feedback services through the Internet. In this study, we only activated the hotel scenario by using the fictitious data. The software we developed consists of two major parts:
  • Data acquisition part
  • Application for data analysis
- 3.1 Data Acquisition
We used subject-to-object bi-partite graph [25 - 26] for the representation of the system trust as shown in Fig. 2 . It describes more efficiently and clearly because, the trust relationships between the object features (contexts) and the subject groups for system trust assessment calculations. In our study, the term “Subject” defines the recommenders that are grouped in five subsets of “Business, Couples, Family, Friends and Solo” under “U (Raters)”. Each recommender gets a unique User ID which remains the same through the updating operations and selects one of these groups in the rating system. The term “Object” means the organization rated on a set of contexts. We assumed three fictitious hotels in our city and created hypothetical data according to the rules of the assessment system. Each hotel with a unique ID is recorded in the assessment system. Assessment Value defines six contexts such as cost, room quality, location, cleanliness, service level, and sleep quality. Recommenders rate the contexts with grades 1 to 5, from the lowest to the highest value. Each rating value has a weight factor chosen by the recommender between 1 and 3 as an importance value “I”. Ratings are also kept with their time stamp value for day, month, year, hour and minute.
PPT Slide
Lager Image
Bi-partite-graph modeling of hotel trust assessment system
In the subject-to-object bipartite graph, U (subject) and V (object) are distinct sets where U = {u 1 , u 2 , u 3 …, u n }; n is the number of elements of U and V ={v 1 , v 2 , v 3 ,…, v m }; m is the number of elements of V. With edges between subject and object, this graph is shown as G = (U(G), V(G), E(G)) where e=( u i ,v j ) ∈E represents a directed edge from vertex u i ∈U to v j ∈V. Each edge is also labeled with a 3-tuple (c, p, t); where “c” is the system trust context, “p” is the system trust metric and “t” is the time specification.
- 3.2 Application for Data Analysis
Hotel values collected from recommenders and processed with importance values are being used to construct the assessment matrix, A_Konak [t 1 ,t 2 ]. This matrix (size of 5 x 6) representing the trust relationship between the recommender’s set U and the context set V of the rated hotel for the time interval [t 1 ,t 2 ] is shown in Fig. 3 . By analyzing the acquired data, the following mathematical descriptions of popularity, trust values and the confidence probability of the hotel objects have been created.
PPT Slide
Lager Image
Assessment matrix example
To provide a normal distribution, we choose the minimum number of recommenders in the group as 30 in the selected time interval. The minimum allowed time interval is 1 month. The model works fine [27] when n≥30. Otherwise, it gives an error message for that recommender group.
In the same time interval, the total ratings given by each recommender group of each context is written in a vector named RC i .
RC 1 = [U 1 V 1 , ..., U 1 V m ] where each element represents the arithmetic sum of the ratings by the recommender group U 1 for each context. By dividing each element of RC 1 by nU 1 (number of recommenders in the group), first row of the assessment matrix is obtained. The process continues for each RC i divided by nU i until i=n. So, other rows of the matrix are sequentially constructed. Other calculations are explained below:
Popularity value for a context is the average rating of all recommender groups in [t 1 ,t 2 ] time interval and can be calculated as;
PPT Slide
Lager Image
For the “n th recommender class” popularity value becomes
PPT Slide
Lager Image
By using Equation (1) and Assessment Matrix in Fig. 3 , popularity values for Hotel Konak can be computed as;
(Cost Context Popularity Value)
PPT Slide
Lager Image
(Room Context Popularity Value)
PPT Slide
Lager Image
Other popularity values for location, cleanliness, service and sleep can also be computed for P V3 , P V4 , P V5 , P V6 .
System trust value is obtained by averaging the context popularity values in [t 1 , t 2 ] time interval
PPT Slide
Lager Image
Being the “m: the number of contexts ” and “k: the highest rating ”, r esulting value is normalized into [0,1] real number interval by dividing it into “m x k”.
By using Equation (2), system trust value of Hotel Konak can be computed as;
PPT Slide
Lager Image
To estimate the accuracy of our system trust value, we use confidence probability. Confidence probability is a pre-selected real value in the interval [0,1].
Confidence probability;
PPT Slide
Lager Image
α : Selected confidence probability . (Confidence probabilities below 0,80 are ignored in our model)
zα value is taken from statistical tables according to the confidence probability [28] from 0,80 to 0,99.
σ : Combined standard deviation of the data collected data for “m contexts”
PPT Slide
Lager Image
It can be obtained from
PPT Slide
Lager Image
n : Number of recommenders in the same group evaluating the same context
x i : Recommendation value of the i th recommender
μ : Arithmetic mean of the recommendations of the group
For the same example, if we select the confidence probability as “ α =0,90”, corresponding z α value is found as 1,65.
If we assume the number of recommenders “n=712” for Hotel Konak in time interval [t 1 ,t 2 ] and the combined standard deviation " α =1,1”; the system trust value for Hotel Konak with confidence probability 0,90 can be computed using Equation (3);
PPT Slide
Lager Image
If we select different “ α ” values;
For α =0.90 and z_ α =1.96, we find [0.582 , 0.744]
For α =0.99 and z_ α =2.58, we find [0.557 , 0.769]
System trust calculations show that as selected confidence probability increases, system trust value interval increases. If the upper bound of system trust value interval is calculated greater than “1”, it is assumed as “1”. If the lower bound of system trust value interval is calculated lesser than “0”, it is assumed as “0”.
4. Trust Assessment Software (TAST)
TAST is a web-based application program using PHP and MySQL environment. An example of TAST web-service can be reached from the link “ ”. Currently, there are four more assessment database that are ready to be used by the recommenders. A sample UML using the case diagram for available user groups is shown in Fig. 4 .
PPT Slide
Lager Image
UML use case diagram of user access rights
Ordinary users have the minimum access rights. They can only select and view the trust information of selected objects. Administrators group is password protected and has full access rights. A maximum of three administrators can be defined for the TAST. Participants can register the survey, answer the survey questions and assess the trust of selected objects. They are the groups of recommenders that actually use the program for rating organizations. During the registration process, they first select a particular user group like Business, Couples, Family, Friends and Solo. Fig. 5 shows a sample snapshot of the calculated assessment matrix.
PPT Slide
Lager Image
Sample snapshot of the calculated assessment matrix
TAST program enables the comparison of trust values of the objects. A comparison of the three objects can be made in a selected time interval. A comparison snapshot of three hotels for the same confidence probability is shown in Fig. 6 .
PPT Slide
Lager Image
Sample comparison snapshot for three hotels
5. Conclusion
In this study, we proposed and applied a formal trust assessment model based on bi-partite graphs. We also developed a web-based software tool, namely TAST (Trust Assessment Software Tool) which collects the ratings of different recommender groups for similar organizations through a web-based user interface. We have assumed three fictitious hotels in our city and created hypothetical data according to the rules of the assessment system since the information of the hotel can easily be utilized to make the necessary trust computations. Bi-partite graphs describes the trust relationships between the recommender groups (Business, Couples, Family, Friends and Solo) that are different customer groups and the related contexts that are common services (cost, room quality, location, cleanliness, service level, and sleep quality) provided by the hotel management. Values were collected from recommenders and processed with predefined importance values to construct the assessment matrix. By analyzing the acquired data, the mathematical descriptions of popularity, trust values and the confidence probability of the hotel objects have been created. The main contributions of this study can be summarized as follows:
• A formal model to assess the trustworthiness of the services provided by the organizations in a specified context-set,
• Use of real-number intervals in trust computations,
• Addition of importance parameter and selected confidence probability,
• Ratings of service features by different customer groups are differentiated in selected time intervals,
• Comparison of trust variations among different organizational service providers,
Our approach gives a more meaningful and realistic approach to the trust assessments between individuals and organizations without debate. As seen from the results, trust value in intervals will cover a larger range when we need higher confidence probability. The explicit formulation of overall trust calculations, including the construction of assessment matrix, popularity and trust values, confidence popularity and related values suggests a straightforward implementation of the model in this hotel environment or any other web services. Evaluation of results determines the satisfaction of different customer groups and the quality of organizational services.
Mahir Kutay graduated from Middle East Technical University, Department of Electric-Electronics in 1980. He received his Ph.D from Dokuz Eylul University in Computer Engineering in 2013. He is a part-time lecturer. His research interests include: Internet Security, Sensor and Embedded Systems, Trust in Web Services, Trust in Cloud Computing.
Tuncay Ercan graduated from Military Academy, Department of Electric-Electronics in 1986. He received his M.S. degree from Ege University, and Ph.D from Dokuz Eylul University in Computer Engineering in 2000 and 2005 respectively. He is an Assistant Professor at Yasar University, Department of Computer Engineering. His research interests include: Advanced Network Technologies, Wireless Ad-hoc and Sensor Networks, Network Security, Network modeling and design. He also studies Educational Technologies and Network Forensics.
Yolum P. , Singh M.P. “An agent based approach for trustworthy service location,” in Proc. of 1st Agents and Peer to Peer Computing Workshop (AP2PC) Bologna, Italy 2003 45 - 56
Svensson G. 2005 “Mutual and interactive trust in business dyads: Condition and process,” European Business Review 17 (5) 411 - 427    DOI : 10.1108/09555340510620320
Breese J.S. , Heckerman D. , Kadie C. “Empirical analysis of predictive algorithms for collaborative filtering,” in Proc. of 14th Conference on Uncertainty in Artificial Intelligence (UAI-98) 1998 43 - 52
Scahafer J.B. , Konstan J. , Riedl J. “Recommender systems in commerce,” Proc. ACM Conference on E-commerce (EC-99) Denver, USA 1999 158 - 166
Huang Z. , Chung W. , Chen H. 2004 “A graph model for e-commerce recommender systems,” Journal of American Society for Information and Science and Technology 55 259 - 274    DOI : 10.1002/asi.10372
Koren Y. , Bell R. , Volinsky C. 2009 “Matrix factorization techniques for recommender systems,” IEEE Computer 42 30 - 37    DOI : 10.1109/MC.2009.263
Schafer J.B. , Frankowski D. , Herlocker J. , Sen S. 2007 “Collaborative filtering recommender systems,” Lecture Notes in Computer Science 4321 291 - 324
Hu Y. , Koren Y. , Volinsky C. “Collaborative filtering for implicit datasets,” in Proc. of 8th IEEE International Conference on Data Mining (ICDM) 2008 263 - 272
Orgun M.A. , Liu J. “Reasoning about dynamics of trust and agent beliefs,” in Proc. of IEEE International Conference on Information Reuse and Integration (IRI) 2006 105 - 110
Carbone M. , Nielsen M. , Sassone V. “A formal model for trust in dynamic networks,” in Proc. of 1st International Conference on Software Engineering and Formal Methods (SEFM’03) 2003 54 - 61
Kuter U. , Goldbeck J. “Sunny: A new algorithm for trust interface in social networks using probabilistic confidence models,” in Proc. of 22nd Conference on Artificial Intelligence (AAAI) 2007 1377 - 1382
Oliveora F. , Pelusoa L. , Romano S. “Refacing: an autonomic approach to network security based on multidimensional trustworthiness,” Computer Networks 52 (14) 2745 - 2763    DOI : 10.1016/j.comnet.2008.04.022
Josang A. , Ismail R. , Bloyd C. 2007 “A Survey of trust and reputation systems for online service provision,” Decision Support Systems 43 (2) 618 - 644    DOI : 10.1016/j.dss.2005.05.019
Aberer K. , Despotovic Z. “Managing trust in a peer-2-peer information system,” in Proc. of Conference on Information and Knowledge Management 2001 310 - 317
Jacobi I. , Kagal L. , Khandewal A. “Rule-based trust assesment on the semantic web,” in Proc. of Conference on Rule-based Reasoning, Autonomous Agents and Multiagent Systems, Programming and Applications 2011 227 - 241
Govindan K. , Wang X. , Khan M. , Dogan G. , Zeng K. , Powell G.M. , Brown T. , Abdelzaher T. , Mohapatra P. “PRONET: Network trust assesment based on incomplete provenance,” in Proc. of Milcom Communications Conference 2011 1213 - 1218
Xin L. , Tredan G. , Datta A. “Metatrust: Discriminant analysis of local information for global trust assesment,” in Proc. of Conference on Autonomous Agents and Multiagent Systems 2011 Vol.3 1071 - 1072
Spanoudakis G. “Dynamic trust assesment of software services,” in Proc. of 2nd International Workshop on Service Oriented Software Engineering 2007 36 - 40
Rettinger A. , Nickles M. , Tresp V. “A statistical relational model for trust learning,” in Proc. of Conf. on Autonomous Agents and Multiagent Systems 2008 763 - 770
Pasternack J. , Roth D. “Comprehensive trust metrics for information networks,” in Proc. of 27th Army Science Conference (ASC) 2010
Pasternack J. , Roth D. “Knowing what to believe (when you already know something),” in Proc. of International Conference on Computational Linguistics (COLING) 2010 877 - 885
Caverlee J. , Liu L. , Webb S. 2010 “The SocialTrust framework for social trust information management: Architecture and algorithms,” Journal of Information Science 180 (1) 95 - 112    DOI : 10.1016/j.ins.2009.06.027
Lee A.J. , Yu T. “Towards a dynamic and composite model of trust,” in Proc. of 14th ACM Symposium on Access Control Models and Technologies 2009 217 - 223
Osman N. , Sierra C. , McNeill F. , Pane J. , Debenham J. 2013 “Trust and matching algorithms for selecting suitable agents,” ACM Transactions on Intelligent Systems and Technology Article no.16 5 (1)    DOI : 10.1145/2542182.2542198
Diestel R. 2000 “Graph Theory” Springer Verlag New York
Yolum P. , Singh M.P. “Service graphs for building trust,” in Proc. of 12th Ontologies, Databases, and Applications Semantics (ODBASE) 2004 509 - 525
Brink D. 2010 “Essentials of Statistics,” Ventus London 47 - 45
Hines W.W. , Montgomery D.C. , Goldsman D.M. , Borror C.M. 2003 “Probability and Statistics in Engineering,” Wiley New York
Marsh S. , Ph.D. Thesis 1994 “Formalising trust as a computational concept,” University of Stirling Ph.D. Thesis
Andersen R. , Borgs C. , Chayes J. , Feige U. “Trust-based recommendation systems: an axiomatic approach,” in Proc. of the 17th international conference on World Wide Web (WWW '08) 2008 199 - 208
Gambetta D. 2008 “Trust: Making and breaking cooperative relations,”Electronic edition, Department of Sociology Bodleian Library, University of Oxford
Malik Z. , Bouguettaya A. 2009 “RATEWeb: Reputation assessment for trust establishment among web services,” Vldb Journal 18 (4) 885 - 911
Liu X. , Yang S. , Guo L. , Shen K. “Profit sharing based routing trust evaluation algorithm in P2P network,” in Proc. of China Grid Annual Conference 2008 352 - 359
Gong S. , Guo Y. , Lei M. , Yang Y. , Liu H. “A dynamic model in P2P services environment for accurate and robust trust evaluation,” ICT and Energy Efficiency and Workshop on Information Theory and Security 2012 56 - 62
Amatrinain X. 2012 “Mining large streams of user data for personalized recommendations,” ACM SIGKDD Newsletter 14 (2) 37 - 48    DOI : 10.1145/2481244.2481250
Bellogin A. , Cantador I. , Diez F. , Castels P. , Chavarriaga E. 2013 “An empirical comparison of social, collaborative filtering and hybrid recommenders,” ACM Transactions on Intelligent Systems and Technology Article no.14 4 (1)    DOI : 10.1145/2414425.2414439
Wu Z. , Weaver A.C. “Application of fuzzy logic in federated trust management for pervasive computing,” in Proc. of Computer Applications and Software Conference 2006 215 - 222